Day 2: Managing Storage

Forenote

Disk = Physical Device
Partition = Section created on disk
Logical Volumes = "Like" partitions but more flexible
Filesystem = Structure of writing to partitions/logical volumes
A Mount Point = Directory
Mount(ing) = Make filesystem available

You mount the filesystem, not the disk or partition

Filesystem Hierarchy Standard (FHS)

Guidelines for the names and locations of files and directories in Linux systems

Some 'static' directories:
/ = root of filesystem (OS structure)
/bin = Command-line binaries
/sbin = Elevated binaries
/boot = Necessary boot files
/dev = Hardware/Software device drivers
/etc = Configuration files
/lib = Shared program libraries
/media = Default mount for removable media
/mnt = Intended for mounting file systems
/var = Logs, default apache root, cache
/opt = Large software packages
/usr = Possible mount as read-only, contained in PATH
/home = User home directories

Default at boot:
/dev = device drivers/temporary filesystem
/proc
/sys

Storage Devices

Physical components that can record and store data

HDDs
SSDs
USBs
External storage drives
Legacy storage drives

Types of File Systems

FAT - Old/Outdated. Will rarely see this.
ext2 - Still supported but outdated. No journal (record of filesystem) capabilities.
ext3 - Supports journal. Only difference between ext2 is the journal.
ext4 - Dynamic inodes. Supports journal.
XFS - High performance journaling. Fast recovery.
NTFS - Windows. Can install support on linux.

Virtual File Systems: Software linking the Linux kernel to other real file systems

smb/samba
cifs
nfs

Identifying devices:
/dev/sda#
blkid - Displays UUID of a device

Inodes

File:

Data
Metadata
inode (address)

List inodes with ls -i

Partitions

Convert large drive into smaller chunks
Must be formatted and assigned a file system first

MBR (Master Boot Record) = First 512 on disk

Can have 4 primary partitions
OR
3 primary with 1 extended
- extended partitions can have multiple logical partitions

Swap Space
Partition on a stoorage device used when the system runs out of physical memory.
Linux pushes unused files from RAM to swap space.

Usually twice the physical RAM size

Best practice is to fill swap with all zeros

Create swap space:
dd if=/dev/zero of=/swapfile bs=1M count=1024

chmod 0600 /swapfile

mkswap /swap

List partitions:
fdisk -l

Make partition changes to disk:

fdisk /dev/sda# 
m

n = Create new partition
w = Write changes
partprobe (updates kernel with new partitions)

fdisk /dev/sda# 
m

n = Create new partition
w = Write changes
partprobe (updates kernel with new partitions)

You can use parted instead of fdisk

Make filesystem on partition:

mkfs -t <ext2,ext3,ext4,XFS> /dev/sda#
OR
mkfs.<ext2,ext3,ext4,XFS> /dev/sda#

/etc/fstab = Filesystems the OS "Knows"

Only root can edit

cat /proc/self/mounts

List mounted filesystems

Mounting Filesystem

mount

Must have a directory to mount to
mount -a for auto mounting filesystems
mount -o don't mount automatically

umount

If you can't unmount, run lsof to list open files that can't be unmounted

Logical Volume Manager

mdadm - RAID

Pete's Victory Looked Fake

Physical : pv commands
- pvdisplay
- pvcreate
- pvcreate <name> /dev/sda1 /dev/sda2 etc...
- pvextend
- pvreduce

Volume Group: vg commands
- vgdisplay
- vgcreate
- vgcreate <name> /dev/sda1 /dev/sda2 etc...
- vgextend <name> /dev/sda3
- vgreduce

Logical Volume: lv commands
- lvdisplay
- lvcreate
- lvcreate <name> <size> <volumegroup>
- lvextend -L200G /dev/mapper/
- lvreduce

- lsblk or blkid list volume blocks

Filesystem: mkfs, xfs_growfs, resize2fs, e2fsck, tune2fs, dumpe2fs
- mkfs.<type> /dev/sda1
- resize2fs <name>
- e2fsck -f <name>
- df -h = Display filesystem usage

MANAGING STORAGE LAB

Creating new partition:

fdisk /dev/sda
m (Help)
p (Print partitions)
n (Create new partition)

Define partition number (remember this)
Define partition start
Enter partition size (+4096M = 4GB)
p (Verify Changes)
w (Write Changes)

partprobe to update kernel

Create a file system on the new partition:
mkfs.xfs /dev/sda4

Create second partition with ext4 file system using parted:
parted /dev/sda

print (Display partitions on /dev/sda)
mkpart (Start partition creation)
Enter Name
Choose file system type (ext4)
Specify start (end of last device from 'print')
Specify end of new partition (Start + ##GB)
print (verify creation)
Create an ext4 filesystem on new partition
- mkfs.ext4 /dev/sda5

Apply Labels to new partitions:
xfs_admin -l /dev/sda4
xfs_admin -L LabelName /dev/sda4
OR
e2label /dev/sda5
e2label /dev/sda5 LabelName

Display Volume Information
ls /dev/mapper

pvscan - Display physical volume information
pvdisplay - More info for physical volumes

vgscan - Display volume groups
vgdisplay - Display information on volume groups

lvscan - Display logical volumes
lvdisplay /logical/volume/path - Display logical volume information

Create physical volumes from backup partitions
pvcreate /dev/sda4 /dev/sda5 - Create physical volumes for each partition
pvdisplay - Verify creation

Create volume group from physical volumes
vgcreate GroupName /dev/sda4 /dev/sda5
vgdisplay - Verify volume group

Create logical volumes in new volume group
lvcreate --name sysbk --size 1GB backup - Create 1GB logical volume named sysbk

Extend data backup volume and reduce log volume
lvextend -L5G /dev/backup/databk - Extends databk volume to 5GB
lvreduce -L1GB /dev/backup/logbk - Reduces logbk volume to 1GB

Create filesystems on logical volumes
mkfs.xfs /dev/backup/sysbk
mkfs.ext4 /dev/backup/databk

Create mount points for logical volumes
mkdir -p /backup/sys /backup/data /backup/log - Creates 3 directories to mount to
mount /dev/backup/sysbk /backup/sys - Mounts logical volume sysbk to directory /backup/sys
mount - Display all mounted volumes
umount /backup/log - Unmount a volume

Mounting logical volumes on boot
vim /etc/fstab

Add /dev/backup/sysbk /backup/sys xfs defaults 0 0 (Logical volume first, then directory)
mount -a - Test /etc/fstab to ensure successful mounting

Get info for block storage devices
lsblk -f

Scan logical volumes for errors
fsck /dev/backup/databk - Must be unounted first with umount
xfs_repair /dev/backup/databk - Repairs the xfs file system

Increasing size of an ext4 file system on a volume
e2fsck -f /dev/backup/databk
lvextend -L6G /dev/backup/databk - Increase logical volume to 6GB
resize2fs /dev/backup/databk - Extend the file ext4 file system on the logical volume

Increase size of an XFS file system on a volume
lvextend -L2G /dev/backup/sysbk
mount /dev/backup/sysbk /backup/sys - Remount file system for next command to work
xfs_growfs /dev/backup/sysbk - Extend XFS filesystem

Identify Storage Space Usage
df -h /backup/data - show filesystems and their used/available blocks
du -h /backup/data- show size of each file in a volume
iostat -d /dev/backup/databk - Show read/write stats of a volume

Enable quotas on a volume
Edit the /etc/fstab file and add ,usrquota next to "defaults".
mount -o /backup/data - remount the filesystem you just altered

Configure quoatas
quotacheck -cugm /backup/data - creates quota files
quotaon -a - enable quotas for all filesystems
edquota -u user1 - set quota limit for user

Create dummy file
dd if=/dev/zero of=/backup/data/myfile bs=## count=##

Check quotas
repquoata /backup/data

Scripting Tips

For Loop
for x in $(cat file); do <command> $x; done

xargs
pritnf '%s\n' oranges apples pears

prints oranges apples pears all on a new line

pritnf '%s\n' oranges apples pears|xargs -I{} groupadd {}

stdin
echo Pa22w0rd|passwd --stdin username

Create Partitions
fdisk
n
+10G

Create Phyiscal Volumes
pvcreate /dev/sda{4..9}

Create Volume Groups
vgcreate vg1 /dev/sda{4..6}
- vgcreate <name> <physical volumes>

Create Logical Volumes
lvcreate --name <name> --size <\size> <vg name>
- lvcreate --name data --size 15GB vg1

Create File System on Logical Volumes
mkfs.ext4 /dev/mapper/vg1-data
mkfs.xfs /dev/mapper/vg1-data

Make directory and mount file system
mkdir /vg1-data
mount /dev/mapper/vg1-data /vg1-data

Make Read-Only without unmounting
mount -o remount,ro /mount/location

Check for inode exhaustin
df -i